This notebook contains a comparative analysis on the task of temporal event ordering between three approaches:
[1] Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy (2013): Learning Whom to Trust with MACE. In: Proceedings of NAACL-HLT 2013.
First we describe the task. Then, we apply the CrowdTruth metrics and give examples of clear and unclear example sentences. We then apply MACE. In the final part we perform two comparisons:
Data*: This notebook uses the data gathered in the "Event Annotation" crowdsourcing experiment published in Rion Snow, Brendan O’Connor, Dan Jurafsky, and Andrew Y. Ng: Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. EMNLP 2008, pages 254–263.
Task Description: Given two events in a text, the crowd has to choose whether the first event happened "strictly before" or "strictly after" the second event. Following, we provide an example from the aforementioned publication:
Text: “It just blew up in the air, and then we saw two fireballs go down to the, to the water, and there was a big small, ah, smoke, from ah, coming up from that”.
Events: go/coming, or blew/saw
A screenshot of the task as it appeared to workers can be seen at the following repository.
The dataset for this task was downloaded from the following repository, which contains the raw output from the crowd on AMT. Currently, you can find the processed input file in the folder named data. Besides the raw crowd annotations, the processed file also contains the sentence and the two events that were given as input to the crowd. However, we have the sentence and the two events only for a subset of the dataset.
In [1]:
# Read the input file into a pandas DataFrame
import pandas as pd
test_data = pd.read_csv("../data/temp.standardized.csv")
test_data.head()
Out[1]:
In [2]:
import crowdtruth
from crowdtruth.configuration import DefaultConfig
Our test class inherits the default configuration DefaultConfig, while also declaring some additional attributes that are specific to the Temporal Event Ordering task:
inputColumns: list of input columns from the .csv file with the input dataoutputColumns: list of output columns from the .csv file with the answers from the workerscustomPlatformColumns: a list of columns from the .csv file that defines a standard annotation tasks, in the following order - judgment id, unit id, worker id, started time, submitted time. This variable is used for input files that do not come from AMT or FigureEight (formarly known as CrowdFlower).annotation_separator: string that separates between the crowd annotations in outputColumnsopen_ended_task: boolean variable defining whether the task is open-ended (i.e. the possible crowd annotations are not known beforehand, like in the case of free text input); in the task that we are processing, workers pick the answers from a pre-defined list, therefore the task is not open ended, and this variable is set to Falseannotation_vector: list of possible crowd answers, mandatory to declare when open_ended_task is False; for our task, this is the list of relationsprocessJudgments: method that defines processing of the raw crowd data; for this task, we process the crowd answers to correspond to the values in annotation_vectorThe complete configuration class is declared below:
In [3]:
class TestConfig(DefaultConfig):
inputColumns = ["gold", "event1", "event2", "text"]
outputColumns = ["response"]
customPlatformColumns = ["!amt_annotation_ids", "orig_id", "!amt_worker_ids", "start", "end"]
# processing of a closed task
open_ended_task = False
annotation_vector = ["before", "after"]
def processJudgments(self, judgments):
# pre-process output to match the values in annotation_vector
for col in self.outputColumns:
# transform to lowercase
judgments[col] = judgments[col].apply(lambda x: str(x).lower())
return judgments
In [4]:
data, config = crowdtruth.load(
file = "../data/temp.standardized.csv",
config = TestConfig()
)
data['judgments'].head()
Out[4]:
In [5]:
results = crowdtruth.run(data, config)
The sentence metrics are stored in results["units"]. The uqs column in results["units"] contains the sentence quality scores, capturing the overall workers agreement over each sentences. The uqs_initial column in results["units"] contains the initial sentence quality scores, before appling the CrowdTruth metrics.
In [6]:
results["units"].head()
Out[6]:
In [7]:
# Distribution of the sentence quality scores and the initial sentence quality scores
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 15, 5
plt.subplot(1, 2, 1)
plt.hist(results["units"]["uqs"])
plt.ylim(0,200)
plt.xlabel("Sentence Quality Score")
plt.ylabel("#Sentences")
plt.subplot(1, 2, 2)
plt.hist(results["units"]["uqs_initial"])
plt.ylim(0,200)
plt.xlabel("Initial Sentence Quality Score")
plt.ylabel("# Units")
Out[7]:
The histograms above show that the final sentence quality scores are nicely distributed, with both lower and high quality sentences. We also observe that, overall, the sentence quality score increased after applying the CrowdTruth metrics, compared to the initial sentence quality scores. While initially more than half of the units had a score of around 0.55, after iteratively applying the CrowdTruth metrics, the majority of the units have quality scores above 0.7.
The sentence quality score is a powerful measure to understand how clear the sentence is and the suitability of the sentence to be used as training data for various machine learning models.
The unit_annotation_score column in results["units"] contains the sentence-annotation scores, capturing the likelihood that an annotation is expressed in a sentence. For each sentence, we store a dictionary mapping each annotation to its sentence-annotation score.
In [8]:
results["units"]["unit_annotation_score"].head()
Out[8]:
First, we sort the sentence metrics stored in results["units"] based on the sentence quality score (uqs), in ascending order. Thus, the most clear sentences are found at the tail of the new structure. Because we do not have initial input for all the units, we first filter these out.
In [9]:
sortedUQS = results["units"].sort_values(["uqs"])
# remove the units for which we don't have the events and the text
sortedUQS = sortedUQS.dropna()
sortedUQS = sortedUQS.reset_index()
We print the most clear unit, which is the last unit in sortedUQS:
In [10]:
sortedUQS.tail(1)
Out[10]:
The following two sentences contain the events that need to be ordered:
Ratners Group PLC's U.S. subsidiary has agreed to acquire jewelry retailer Weisfield's Inc.
Ratners and Weisfield's said they reached an agreement in principle for the acquisition of Weisfield's by Sterling Inc.
The unit is very clear because the second sentence clearly states that before acquiring Weisfield's Inc, the two parts reached an agreement, which means that acquire happened after reached.
In [11]:
print("Text: %s" % sortedUQS["input.text"].iloc[len(sortedUQS.index)-1])
print("\n Event1: %s" % sortedUQS["input.event1"].iloc[len(sortedUQS.index)-1])
print("\n Event2: %s" % sortedUQS["input.event2"].iloc[len(sortedUQS.index)-1])
print("\n Expert Answer: %s" % sortedUQS["input.gold"].iloc[len(sortedUQS.index)-1])
print("\n Crowd Answer with CrowdTruth: %s" % sortedUQS["unit_annotation_score"].iloc[len(sortedUQS.index)-1])
print("\n Crowd Answer without CrowdTruth: %s" % sortedUQS["unit_annotation_score_initial"].iloc[len(sortedUQS.index)-1])
We use the same structure as above and we print the most unclear unit, which is the first unit in sortedUQS:
In [12]:
sortedUQS.head(1)
Out[12]:
The following sentence contains the events that need to be ordered:
Magna International Inc..'s chief financial officer, James McAlpine, resigned and its chairman, Frank Stronach, is stepping in to help turn the automotive-parts manufacturer around, the company said.
The unit is unclear due to various reasons. First of all, the sentence is very long and difficult to read. Second, there is a series of events mentioned in the text and third, it is not very clearly stated if the "turning" event is happening prior or after the "announcement".
In [13]:
print("Text: %s" % sortedUQS["input.text"].iloc[0])
print("\n Event1: %s" % sortedUQS["input.event1"].iloc[0])
print("\n Event2: %s" % sortedUQS["input.event2"].iloc[0])
print("\n Expert Answer: %s" % sortedUQS["input.gold"].iloc[0])
print("\n Crowd Answer with CrowdTruth: %s" % sortedUQS["unit_annotation_score"].iloc[0])
print("\n Crowd Answer without CrowdTruth: %s" % sortedUQS["unit_annotation_score_initial"].iloc[0])
The worker metrics are stored in results["workers"]. The wqs columns in results["workers"] contains the worker quality scores, capturing the overall agreement between one worker and all the other workers. The wqs_initial column in results["workers"] contains the initial worker quality scores, before appling the CrowdTruth metrics.
In [14]:
results["workers"].head()
Out[14]:
In [15]:
# Distribution of the worker quality scores and the initial worker quality scores
plt.rcParams['figure.figsize'] = 15, 5
plt.subplot(1, 2, 1)
plt.hist(results["workers"]["wqs"])
plt.ylim(0,30)
plt.xlabel("Worker Quality Score")
plt.ylabel("#Workers")
plt.subplot(1, 2, 2)
plt.hist(results["workers"]["wqs_initial"])
plt.ylim(0,30)
plt.xlabel("Initial Worker Quality Score")
plt.ylabel("#Workers")
Out[15]:
The histograms above shows the worker quality scores and the initial worker quality scores. We observe that the worker quality scores are distributed across a wide spectrum, from low to high quality workers. Furthermore, the worker quality scores seem to have, overall, improved after computing the CrowdTruth iterations, compared to the initial worker quality scores, which indicates that the difficulty of the units was taken into consideration.
Low worker quality scores can be used to identify spam workers, or workers that have misunderstood the annotation task. Similarly, high worker quality scores can be used to identify well performing workers.
The annotation metrics are stored in results["annotations"]. The aqs column contains the annotation quality scores, capturing the overall worker agreement over one annotation.
In [16]:
results["annotations"]
Out[16]:
In the dataframe above we observe that after iteratively computing the sentence quality scores and the worker quality scores the overall agreement on the annotations increased. This can be seen when comparing the annotation quality scores with the initial annotation quality scores.
We first pre-processed the crowd results to create compatible files for running the MACE tool. Each row in a csv file should point to a unit in the dataset and each column in the csv file should point to a worker. The content of the csv file captures the worker answer for that particular unit (or remains empty if the worker did not annotate that unit).
The following implementation of MACE has been used in these experiments: https://github.com/dirkhovy/MACE.
In [16]:
# MACE input file sample
import numpy as np
mace_test_data = pd.read_csv("../data/mace_temp.standardized.csv", header=None)
mace_test_data = mace_test_data.replace(np.nan, '', regex=True)
mace_test_data.head()
Out[16]:
For each sentence and each annotation, MACE computes the sentence annotation probability score, which shows the probability of each annotation to be expressed in the sentence. MACE sentence annotation probability score is similar to the CrowdTruth sentence-annotation score.
In [17]:
# MACE sentence annotation probability scores:
import pandas as pd
mace_data = pd.read_csv("../data/results/mace_units_temp.csv")
mace_data.head()
Out[17]:
For each worker in the annotators set we have MACE worker competence score, which is similar to the CrowdTruth worker quality score.
In [18]:
# MACE worker competence scores
mace_workers = pd.read_csv("../data/results/mace_workers_temp.csv")
mace_workers.head()
Out[18]:
We read the worker quality scores as returned by CrowdTruth and MACE and merge the two dataframes
In [19]:
mace_workers = pd.read_csv("../data/results/mace_workers_temp.csv")
crowdtruth_workers = pd.read_csv("../data/results/crowdtruth_workers_temp.csv")
workers_scores = pd.merge(mace_workers, crowdtruth_workers, on='worker')
workers_scores = workers_scores.sort_values(["wqs"])
workers_scores.head()
Out[19]:
Plot the quality scores of the workers as computed by both CrowdTruth and MACE:
In [20]:
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.scatter(
workers_scores["competence"],
workers_scores["wqs"],
)
plt.plot([0, 1], [0, 1], 'red', linewidth=1)
plt.title("Worker Quality Score")
plt.xlabel("MACE")
plt.ylabel("CrowdTruth")
Out[20]:
In the plot above we observe that MACE and CrowdTruth have quite similar worker quality scores. It seems, however, that MACE favours extreme values, which means that the identified low quality workers will have very low scores, e.g., very close to 0.0 and the best workers will have quality scores of 1.0, or very close to 1.0. On the other side, CrowdTruth has a smaller interval of values, starting from around 0.1 to 0.9.
Following, we compute the correlation between the two values using Spearman correlation and Kendall's tau correlation, to see whether the two values are correlated. More exactly, we want to see whether, overall, both metrics identify as low quality or high quality similar workers, or they are really divergent in their outcome.
In [21]:
from scipy.stats import spearmanr
x = workers_scores["wqs"]
x_corr = workers_scores["competence"]
corr, p_value = spearmanr(x, x_corr)
print ("correlation: ", corr)
print ("p-value: ", p_value)
Spearman correlation shows shows a very strong correlation between the two computed values, and the correlation is significant. This means that overall, even if the two metrics provide different values, they are indeed correlated and low quality workers receive low scores and high quality workers receive higher scores from both aggregation methods.
In [22]:
from scipy.stats import kendalltau
x1 = workers_scores["wqs"]
x2 = workers_scores["competence"]
tau, p_value = kendalltau(x1, x2)
print ("correlation: ", tau)
print ("p-value: ", p_value)
Even with Kendall's tau rank correlation, we observe a strong correlation between the two computed values, where the correlation is significant. This means that the aggregation methods, MACE and CrowdTruth rank the workers based on their quality in a similar way.
Further, we compute the difference of the two quality scores and we check one worker for which the difference is very high.
In [23]:
workers_scores["diff"] = workers_scores["wqs"] - workers_scores["competence"]
workers_scores = workers_scores.sort_values(["diff"])
workers_scores.tail(5)
Out[23]:
We take for example the worker with the id "A2KONK3TIL5KVX" and check the overall disagreement among the workers on the units annotated by them. MACE rated the worker with a quality score of 0.002 while CrowdTruth rated the worker with a higher quality score of 0.32.
What we observe in the dataframe below, where we show the units annotated by the worker "A2KONK3TIL5KVX", is that the worker "A2KONK3TIL5KVX" annotated, in general, units with high disagreement, i.e., which are not very clear. While MACE marked the worker as low quality because it seems that they always picked the same answer, CrowdTruth also considered the difficulty of the units, and thus, giving it a higher weight.
In [24]:
units = list(test_data[test_data["!amt_worker_ids"] == "A2KONK3TIL5KVX"]["orig_id"])
all_results = results["units"].reset_index()
units_df = all_results[all_results["unit"].isin(units)]
units_df = units_df.sort_values(["uqs_initial"])
units_df.head(10)
Out[24]:
Next, we look into the crowd performance in terms of F1-score compared to expert annotations. We compare the crowd performance given the three aggregation methods: CrowdTruth, MACE and Majority Vote.
In [25]:
mace = pd.read_csv("../data/results/mace_units_temp.csv")
crowdtruth = pd.read_csv("../data/results/crowdtruth_units_temp.csv")
The following two functions compute the F1-score of the crowd compared to the expert annotations. The first function computes the F1-score at every sentence-annotation score threshold. The second function computes the F1-score for the majority vote approach, i.e., when at least half of the workers picked the answer.
In [26]:
def compute_F1_score(dataset, label, gold_column, gold_value):
nyt_f1 = np.zeros(shape=(100, 2))
for idx in xrange(0, 100):
thresh = (idx + 1) / 100.0
tp = 0
fp = 0
tn = 0
fn = 0
for gt_idx in range(0, len(dataset.index)):
if dataset[label].iloc[gt_idx] >= thresh:
if dataset[gold_column].iloc[gt_idx] == gold_value:
tp = tp + 1.0
else:
fp = fp + 1.0
else:
if dataset[gold_column].iloc[gt_idx] == gold_value:
fn = fn + 1.0
else:
tn = tn + 1.0
nyt_f1[idx, 0] = thresh
if tp != 0:
nyt_f1[idx, 1] = 2.0 * tp / (2.0 * tp + fp + fn)
else:
nyt_f1[idx, 1] = 0
return nyt_f1
def compute_majority_vote(dataset, label, gold_column, gold_value):
tp = 0
fp = 0
tn = 0
fn = 0
for j in range(len(dataset.index)):
if dataset[label].iloc[j] >= 0.5:
if dataset[gold_column].iloc[j] == gold_value:
tp = tp + 1.0
else:
fp = fp + 1.0
else:
if dataset[gold_column].iloc[j] == gold_value:
fn = fn + 1.0
else:
tn = tn + 1.0
return 2.0 * tp / (2.0 * tp + fp + fn)
F1-score for the annotation "before":
In [27]:
F1_crowdtruth = compute_F1_score(crowdtruth, "before", "gold", "before")
print("Best CrowdTruth F1 score for annotation 'before': ", F1_crowdtruth[F1_crowdtruth[:,1].argsort()][-1:])
F1_mace = compute_F1_score(mace, "before", "gold", "before")
print("Best MACE F1 score for annotation 'before': ", F1_mace[F1_mace[:,1].argsort()][-1:])
F1_majority_vote = compute_majority_vote(crowdtruth, 'before_initial', "gold", "before")
print("Majority Vote F1 score for annotation 'before': ", F1_majority_vote)
F1-score for the annotation "after":
In [28]:
F1_crowdtruth = compute_F1_score(crowdtruth, "after", "gold", "after")
print("Best CrowdTruth F1 score for annotation 'after': ", F1_crowdtruth[F1_crowdtruth[:,1].argsort()][-1:])
F1_mace = compute_F1_score(mace, "after", "gold", "after")
print("Best MACE F1 score for annotation 'after': ", F1_mace[F1_mace[:,1].argsort()][-1:])
F1_majority_vote = compute_majority_vote(crowdtruth, 'after_initial', "gold", "after")
print("Majority Vote F1 score for annotation 'after': ", F1_majority_vote)
From the results above we observe that MACE and CrowdTruth perform very close to each other, and they both perform a bit better than Majority Vote, but not significantly better. As we can observe in the overall initial sentence quality score, there aren't that many unclear sentences in the dataset where half of the workers picked "true" as an answer and half as "false" (less than 60 examples out of 462).
To further explore the CrowdTruth and MACE quality metrics, download the aggregation results in .csv format for: